Comparative Analysis of SDG Implementation Evolution Worldwide

1 Introduction

1.1 Overview and Motivation

The global significance of the SDGs is our basis. The adoption of the SDGs by the United Nation in 2015 marked a significant global commitment to address pressing issues such as poverty, inequality, climate, change, and more. The fact that these goals were unanimously adopted by 193 member states underscores their importance. This prompted us to ask ourselves, can we evaluate the progress? What has really been done so far? Although the SDGs have attracted considerable attention and backing, it is essential to evaluate the events preceding and following their implementation. Understanding the actions taken and progress made is essential in determining if these global commitments are resulting in tangible enhancements to individuals’ lives. By examining the evolution of all countries and their respective contributions towards achieving the SDGs, we can develop a comprehensive understanding of collective efforts and identify potential disparities or gaps.

1.2 Related Work

Given the scale and the challenge that revolove around the SDGs, there are many researches that have been done on the topic. We have found some interesting ones that we found interesting to look at to get a better understanding of the topic. One of them, is none other than the report of the United Nations on the SDGs. This report is a comprehensive annual assessment of global and regional progress towards the SDGs. It is based on the latest available data and is prepared by the Statistics Division of the United Nations Department of Economic and Social Affairs. Find below a list of other interesting researches on the topic:

• The Sustainable Development Goals Report 2023
• Initial progress in implementing the Sustainable Development Goals (SDGs): a review of evidence from countries

1.3 Research questions

1. Focus on factors: What can explain the state of the countries regarding sustainable development? (we will analyse different factors: scores from the human freedom index, GDP per capita, military expenditures in % of GDP/government expenditure, unemployment rate, internet usage). See data description for more precise information about the factors.

2. Focus on time: How has the adoption of the SDGs in 2015 influenced the achievement of SDGs? (we want to compare the achievement (SDG scores: there are scores calculated even before the adoption) of the different countries before and after 2015 to see if the adoption of SDG gave a real “push” to sustainable development)

3. Focus on events: Is the evolution in sustainable development influenced by uncontrollable events, such as economic crisis, health crises and natural disasters? (we will analyse the impact of the COVID, natural disasters and conflicts (# deaths, damages, etc.) on the SDG scores). See data description for more precise information about how the impact of these events are materialized into data.

4. Focus on relationship between SDGs: How are the different SDGs linked? (We want to see if some SDGs are linked in the fact that a high score on one implies a high score on the other, and thus if we can make groups of SDGs that are comparable in that way).

2 Data

2.1 Sources

We are collecting our Data from the sustainability development report (SDG), the international labour organization (ILOSTAT), the World Bank, Our world in data, the CATO institute, one from Kaggle (disasters: we couldn’t find relevant accessible information from somewhere else) and GitHub. We found different datasets containing useful information in relation with the SDGs. The details about these data and the links are presented in the next section. Utilizing the kableExtra package, we provide a comprehensive list and corresponding links to our sources, as outlined below:

Name of the Table	Source
D1_1_SDG	dashboards.sdgindex.org
D2_2_Unemployment_rate	ilo.org
D3_0_GDP_per_capita	data.worldbank.org
D3_1_Military_expenditure_percent_GDP	data.worldbank.org
D3_2_Military_expenditure_percent_gov_exp	data.worldbank.org
D4_0_Internet_usage	ourworldindata.org
D5_0_Human_freedom_index	cato.org
D6_0_Disaters	kaggle.com
D7_0_COVID	github.com
D8_0_Conflicts	datacatalog.worldbank.org

2.2 Description

During the wrangling process, we added data to our table (D1_1_SDG) from different other datasets and match them based on the country code, and the year. The tables below show all the variables present in our 9 databases. We will then merge them to have our final table for the analysis.

2.2.1 Our databases

Sustainable Development Goals database (DGD1_1_SDG)

The Sustainable Development Goals (SDGs) are a universal set of 17 interlinked goals that were adopted by the United Nations in 2015 as part of the 2030 Agenda for Sustainable Development. These goals provide a shared blueprint for peace and prosperity for people and the planet, now and into the future.

Source: UN Communication materials

Our primary database focuses on the Sustainable Development Goals (SDG). Below is a table summarizing the key variables included:

Variable Name	Explanation
code	Country code (ISO)
country	Country name
year	Year of the observation (2000-2022)
overallscore	Overall score on all 17 SDGs (the score are % of achievement of the goals determined by the UN based on several indicators)
goal1:goal17	Score on each SDG except SDG 14 (16 variables)
population	Population of the country

Unemployment rate database (D2_2_Unemployment_rate)

This database give us comprehensive data on the unemployment rates for each country from 2000 to 2022. Originally, it included categories based on various age groups. However, for simplicity and coherence, the database has been streamlined to focus exclusively on the unemployment rates of individuals aged 15 years and older.

Variable Name	Explanation
code	Country code (ISO)
country	Country name
year	Year of the observation (2000-2022)
unemployment.rate	Unemployment rate (% of the population 15 years old and older)

GDP per capita database (D3_0_GDP_per_capita)

This database offers detailed information on the GDP per capita in dollars for various countries, covering the period from 2000 to 2022. It is designed to provide insights into the economic performance of each country over these years, measured through the lens of per capita GDP.

Variable Name	Explanation
code	Country code (ISO)
country	Country name
year	Year of the observation (2000-2022)
GDPpercapita	GDP per capita

Proportion of the GDP dedicated to Military expenditures database (D3_1_Military_expenditure_percent_GDP)

This database provides an insightful view of the proportion of their respective GDPs that countries have allocated to military expenditures. It covers the period from 2000 to 2022.

Variable Name	Explanation
code	Country code (ISO)
country	Country name
year	Year of the observation (2000-2022)
MilitaryExpenditurePercentGDP	Military expenditures in percentage of GDP

Internet usage database (D4_0_Internet_usage)

This database provides information on the percentage of the population that uses the internet in each country. It covers the period from 2000 to 2022.

Variable Name	Explanation
code	Country code (ISO)
country	Country name
year	Year of the observation (2000-2022)
internet.usage	Internet usage (% of the population)

Human freedom index database (D5_0_Human_freedom_index)

This database provides information on the Human Freedom Index (HFI) for each country. The HFI is a composite index that measures the degree to which people are free to enjoy important rights and freedoms. It covers the period from 2000 to 2022.

Variable Name	Explanation
code	Country code (ISO)
country	Country name
year	Year of the observation (2000-2022)
region	Part of the world, group of countries (e.g. Eastern Europe, Dub-Saharan Africa, South Asia, etc.)
pf_law	Rule of law, mean score of: Procedural justice, Civil, justice, Criminal justice, Rule of law (V-Dem)
pf_security	Security and safety, mean score of: Homicide, Disappearances conflicts, terrorism
pf_movement	Freedom of movement (V-Dem), Freedom of movement (CLD)
pf_religion	Freedom of religion, Religious organization, repression
pf_assembly	Civil society entry and exit, Freedom of assembly, Freedom to form/run political parties, Civil society repression
pf_expression	Direct attacks on the press, Media and expression (V-Dem), Media and expression (Freedom House), Media and expression (BTI), Media and expression (CLD)
pf_identity	Same-sex relationships, Divorce, Inheritance rights, Female genital mutilation
ef_gouvernment	Government consumption, Transfers and subsidies, Government investment, Top marginal tax rate, State ownership of assets
ef_legal	Judicial independence, Impartial courts, Protection of property rights, Military interference Integrity of the legal system Legal enforcementof contracts, Regulatory costs, Reliability of police
ef_money	Money growth, Standard deviation of inflation, Inflation: Most recent year, Freedom to own foreign currency
ef_trade	Tariffs, Regulatory trade barriers, Black-market exchange rates, Movement of capital and people
ef_regulation	Credit market regulations, Labor market regulations, Business regulations

Disaster list database (D6_0_Disaters)

This database provides information on the number of deaths, injured, affected and homeless people, as well as the total number of affected people and the total of infrastructure damages caused by disasters in each country. It covers the period from 2000 to 2021.

Variable Name	Explanation
code	Country code (ISO)
country	Country name
year	Year of the observation (2000-2021)
continent	Continents touched by the disasters such as floods, ouragan
total_deaths	Number of deaths caused by disasters
no_injured	Number of injured caused by disasters
no_affected	Number of affected caused by disasters
no_homeless	Number of homeless caused by disasters
total_affected	Total number of affected caused by disasters
total_damages	Total of infrastructure damages

COVID database (D7_0_COVID)

This database provides information on the number of people dead due to COVID, the number of COVID cases and the Government Response Stringency Index in each country. It covers the period from 2020 to 2022.

Variable Name	Explanation
code	Country code (ISO)
country	Country name
year	Year of the observation (2020-2022)
deaths_per_million	Number of people dead due to COVID
cases_per_million	Number of COVID cases
stringency	Government Response Stringency Index: composite measure based on 9 response indicators including school closures, workplace closures, and trave

Conflicts database (D8_0_Conflicts)

This database provides information on the number of deaths, the number of people affected and the maximum intensity of conflicts in each country. It covers the period from 2000 to 2022.

Variable Name	Explanation
code	Country code (ISO)
country	Country name
year	Year of the observation (2000-2022)
ongoing	Variable coded 1 for more than 25 deaths in intrastate conflict and 0 otherwise according to UCDP/PRIO Armed Conflict Dataset 17.1.
sum_deaths	Best estimate of deaths in all categories of violence (non-state, one-sided and state-based) recorded by the Uppsala Conflict Data Program in the country based on the UCDP GED dataset (unpublished 2016 data). The location of these events is used for estimating the extent of violence.
pop_affected	Share of population affected by violence in percentage (0 to 100) measured as described above based on population data from CIESIN, the PRIO-GRID structure as well as UCDP GED.
area_affected	Area affected by conflict
maxintensity	Two different intensity levels are coded: minor armed conflicts (1) and wars (2), Takes the max intensity of conflict in the country so that it is coded 2 if there is at least one war (>=1000 deaths in intrastate conflict) during the year. Data from UCDP/PRIO Armed Conflict Dataset 17.1.

2.3 Wrangling/cleaning

To accommodate the large scale of the datasets, we pre-cleaned each one prior to merging. This streamlined the process, simplifying the cleaning of the final, combined dataset. The treatment of missing values wil be taken care of after merging our datasets.

2.3.1 Dataset on SDG

This is our main dataset, that we clean in order to keep the columns containing the following information: country name, country code, year, population, overall score and the SDGs scores.

We start by importing the data and converting it into a DataFrame. Next, we rename the columns and convert the scores into numeric variables.

Code

D1_0_SDG <- read.csv(here("scripts","data","SDG.csv"), sep = ";")
D1_0_SDG <- as.data.frame(D1_0_SDG)

D1_0_SDG <- D1_0_SDG[,1:22]

colnames(D1_0_SDG) <- c("code", "country", "year", "population",
                        "overallscore", "goal1", "goal2", "goal3",
                        "goal4", "goal5", "goal6", "goal7", "goal8",
                        "goal9", "goal10", "goal11", "goal12",
                        "goal13", "goal14", "goal15", "goal16",
                        "goal17")

D1_0_SDG[["overallscore"]] <- as.double(gsub(",", ".", D1_0_SDG[["overallscore"]]))

makenumSDG <- function(D1_0_SDG) {
  for (i in 1:17) {
    varname <- paste("goal", i, sep = "")
    D1_0_SDG[[varname]] <- as.double(gsub(",", ".", D1_0_SDG[[varname]]))
  }
  return(D1_0_SDG)
}

D1_0_SDG <- makenumSDG(D1_0_SDG)

We proceed by examining the missing values.

Code

propmissing <- numeric(length(D1_0_SDG))

for (i in 1:length(D1_0_SDG)){
  proportion <- mean(is.na(D1_0_SDG[[i]]))
  propmissing[i] <- proportion
}
variable_names <- colnames(D1_0_SDG)
 
prop_missing_data <- data.frame(variable = variable_names, prop_missing = propmissing)

ggplot(prop_missing_data, aes(x = variable, y = prop_missing)) +
   geom_bar(stat = "identity", fill = "skyblue", color = "black") +
   labs(title = "NAs by columns in the main dataset",
        x = "Variable",
        y = "Proportion of Missing Values") +
   theme_minimal()+
   coord_flip()

Observing that the ‘population’ column contains numerous NAs, we investigate and discover that missing values are common, as some observations represent regions, not countries. Therefore, we can safely exclude these observations.

Code

SDG0 <- D1_0_SDG %>%
  group_by(code) %>%
  select(population) %>%
  summarize(NaPop = mean(is.na(population))) %>%
  filter(NaPop != 0)

ggplot(SDG0, aes(x = code, y = NaPop)) +
  geom_bar(stat = "identity", fill = "lightgreen", color = "black") +
  labs(title = "NAs by row in population variable are for regions and not countries",
       x = "Code",
       y = "Proportion of Missing Values") +
  theme_minimal() +
  theme(axis.text.x = element_text(angle = 45, hjust = 1))

D1_0_SDG <- D1_0_SDG %>%
  filter(!str_detect(code, "^_"))

Now, there are no missing values in the ‘population’ variable, and we observe that it contains information on 166 countries.

We notice that NAs are present in only three SDG scores: 1, 10, and 14. Additionally, when a country has NAs, they occur across all years or not at all. Consequently, we decide to conduct further investigations on these three SDG scores to determine whether to include them in our analysis.

For goal 1, there are only 9.04% missing values in 15 different countries. Goal 1 being “End poverty”, we decide to keep it and only remove the countries with no information for the analysis.

Code

SDG2 <- D1_0_SDG |> 
  group_by(code) |> 
  select(contains("goal")) |> 
  summarize(Na1 = mean(is.na(goal1))) |>
  filter(Na1 != 0)
country_number <- length(unique(D1_0_SDG$country))
length(unique(SDG2$code))/country_number
#> [1] 0.0904

For goal 10, there are only 10.2% missing values in 17 different countries. Goal 10 being “reduced inequalities”, we decide to keep it and only remove the countries with no information for the analysis.

Code

SDG3 <- D1_0_SDG |> 
  group_by(code) |> 
  select(contains("goal")) |> 
  summarize(Na10 = mean(is.na(goal10))) |>
  filter(Na10 != 0)

length(unique(SDG3$code))/country_number
#> [1] 0.102

For goal 14, there are 24.1% missing values in 40 different countries. Goal 14 being “life under water”, we decide not to keep it, because other SDG such as “life on earth” and “clean water” already treat similar subjects.

Code

SDG4 <- D1_0_SDG |> 
  group_by(code) |> 
  select(contains("goal")) |> 
  summarize(Na14 = mean(is.na(goal14))) |>
  filter(Na14 != 0)

length(unique(SDG4$code))/country_number
#> [1] 0.241

D1_0_SDG <- D1_0_SDG %>% select(-goal14)

We will work with various datasets and merge them using the country code and year as key identifiers. To ensure accurate matching, we first verify that country names are encoded in UTF-8 format. Then, we standardize the names of the countries (requiring a custom match for Turkey) and the country codes, utilizing the countrycode library. Additionally, we compile a list of all country codes from the main database to filter the other datasets. Lastly, we complete the database to include all possible “country, year” combinations, ensuring the total number of rows remains unchanged.

Code

D1_0_SDG$country <- stri_encode(D1_0_SDG$country, to = "UTF-8")

D1_0_SDG <- D1_0_SDG %>%
  mutate(country = countrycode(country, "country.name", "country.name", custom_match = c("T�rkiye"="Turkey")))

D1_0_SDG$code <- countrycode(
  sourcevar = D1_0_SDG$code,
  origin = "iso3c",
  destination = "iso3c",
)

list_country <- c(unique(D1_0_SDG$code))

D1_0_SDG_country_list <- D1_0_SDG %>%
  filter(code %in% list_country) %>%
  select(code, country)

D1_0_SDG_country_list <- D1_0_SDG_country_list %>%
  select(code, country) %>%
  distinct()

Finally, we complete the database to ensure there are no missing pairs of (year, code).

Here are the first few lines of the cleaned dataset on SDG achievement scores:

<<<<<<< Updated upstream

=======

>>>>>>> Stashed changes

For this first dataset, we reduced the size from 4,140 observations across 120 variables to 3,818 observations for 21 variables.

As said, this is now our main dataset. All subsequent datasets will be merged with this dataset. Therefore, for all the following datasets, we will make sure that we only keep data for the same countries and years as in this dataset. We have a total of 166 countries and the years range from 2000 to 2022.

2.3.2 Dataset on Unemployment rate

In this dataset, the initial step involves importing the data. Next, we ensure that the names and codes of the countries are formatted in UTF-8, preventing any discrepancies due to mismatches in country names. Following this, we modify the column names and filter the data to include only the relevant countries and years, specifically the years 2000 to 2022, covering 166 countries from our primary dataset.

Code

D2_1_Unemployment_rate <- read.csv(here("scripts","data","UnemploymentRate.csv")) %>%
  as.data.frame() %>%
  mutate(
    country = iconv(ref_area.label, to = "UTF-8", sub = "byte"),
    country = countrycode(country, "country.name", "country.name"),
    year = time,
    `unemployment rate` = obs_value / 100,
    age_category = classif1.label,
    sex = sex.label
  ) %>%
  select(-ref_area.label, -time, -obs_value, -classif1.label, -sex.label, -source.label, -obs_status.label, -indicator.label) %>%
  merge(D1_0_SDG_country_list[, c("country", "code")], by = "country", all.x = TRUE) %>%
  filter(year >= 2000 & year <= 2022,
         !str_detect(sex, fixed("Male")) & !str_detect(sex, fixed("Female")),
         code %in% D1_0_SDG_country_list$code,
         age_category == "Age (Youth, adults): 15+") %>%
  select(code, country, year, `unemployment rate`) %>%
  distinct()

Here are the first few lines of the cleaned dataset on Unemployment rate:

<<<<<<< Updated upstream

=======

>>>>>>> Stashed changes

For this first dataset, we reduced the size from 82,800 observations across 8 variables to 3,812 observations for 5 variables.

2.3.3 Dataset on GDP military Expenditures

We have three different databases which contain information on each countries over the years. Each year represent one variable. We want to extract three variables for our analysis: GDP per capita, military expenditures in percentage of the GDP and military expenditures in percentage of government expenditures.

Code

GDPpercapita <-
  read.csv(here("scripts","data","GDPpercapita.csv"), sep = ";")
MilitaryExpenditurePercentGDP <-
  read.csv(here("scripts","data","MilitaryExpenditurePercentGDP.csv"), sep = ";")
MiliratyExpenditurePercentGovExp <-
  read.csv(here("scripts","data","MiliratyExpenditurePercentGovExp.csv"), sep = ";")

After importing the data, we fill in the missing country codes using the column Indicator.Name, because we realized after some manipulations, that some of the country codes were false, but the next column contained the right ones.

Code

fill_code <- function(data){
  data <- data %>%
    mutate(Country.Code = ifelse(!grepl("^[A-Z]{3}$", Country.Code), Indicator.Name, Country.Code))
}

We create a set of functions that we will apply to each database. First, remove the variables that we don’t need, which are the years before 2000. Second, make sure that the values are numeric and rename the year variables (because they all had an “X” before year number). Third, transform the database from wide to long, in order to match the main database. Fourth, transform the year variable into an integer variable and rearrange and rename the columns to match the ones of the other databases. Then, we apply these transformations to the three databases.

Code

remove <- function(data){
  years <- seq(1960, 1999)
  removeyears <- paste("X", years, sep = "")
  data <- data[, !(names(data) %in% c("Indicator.Name", "Indicator.Code", "X", removeyears))]
}

makenum <- function(data) {
  for (i in 2000:2022) {
    year <- paste("X", i, sep = "")
    data[[year]] <- as.numeric(data[[year]])
  }
  return(data)
}

renameyear <- function(data) {
  for (i in 2000:2022) {
    varname <- paste("X", i, sep = "")
    names(data)[names(data) == varname] <- gsub("X", "", varname)
  }
  return(data)
}

wide2long <- function(data) {
  data <- pivot_longer(data, 
                       cols = -c("Country.Name", "Country.Code"), 
                       names_to = "year", 
                       values_to = "data")
  return(data)
}

yearint <- function(data) {
  data$year <- as.integer(data$year)
  return(data)
}

nameorder <- function(data) {
  colnames(data) <- c("country", "code", "year", "data")
  data <- data %>% select(c("code", "country", "year", "data"))
}

cleanwide2long <- function(data){
  data <- fill_code(data)
  data <- remove(data)
  data <- makenum(data)
  data <- renameyear(data)
  data <- wide2long(data)
  data <- yearint(data)
  data <- nameorder(data)
}

GDPpercapita <- cleanwide2long(GDPpercapita)
MilitaryExpenditurePercentGDP <- cleanwide2long(MilitaryExpenditurePercentGDP)
MiliratyExpenditurePercentGovExp <- cleanwide2long(MiliratyExpenditurePercentGovExp)

We rename the colums with the main information, standardize the country code and remove the countries that are not in our main database. We see that all the 166 countries are there.

Code

GDPpercapita <- GDPpercapita %>%
  rename(GDPpercapita = data)
MilitaryExpenditurePercentGDP <- MilitaryExpenditurePercentGDP %>%
  rename(MilitaryExpenditurePercentGDP = data)
MiliratyExpenditurePercentGovExp <- MiliratyExpenditurePercentGovExp %>%
  rename(MiliratyExpenditurePercentGovExp = data)

GDPpercapita$code <- countrycode(
  sourcevar = GDPpercapita$code,
  origin = "iso3c",
  destination = "iso3c",
)

MilitaryExpenditurePercentGDP$code <- countrycode(
  sourcevar = MilitaryExpenditurePercentGDP$code,
  origin = "iso3c",
  destination = "iso3c",
)

MiliratyExpenditurePercentGovExp$code <- countrycode(
  sourcevar = MiliratyExpenditurePercentGovExp$code,
  origin = "iso3c",
  destination = "iso3c",
)

GDPpercapita <- GDPpercapita %>% filter(code %in% list_country)
length(unique(GDPpercapita$code))
#> [1] 166

MilitaryExpenditurePercentGDP <- MilitaryExpenditurePercentGDP %>% filter(code %in% list_country)
length(unique(MilitaryExpenditurePercentGDP$code))
#> [1] 166

MiliratyExpenditurePercentGovExp <- MiliratyExpenditurePercentGovExp %>% filter(code %in% list_country)
length(unique(MiliratyExpenditurePercentGovExp$code))
#> [1] 166

There were only 157 countries that were both in the main SDG dataset and in these 3 datasets, but we suspected that some of the missing countries were in the database but not rightly matched. Indeed, Bahamas was in the database but instead of the code “BHS” there was “The”, for “COD” it was “Dem. Rep.”, for “COG” it was “Rep”, etc. We remarked that the code is in another column of the initial database: “Indicator.Name”. We went back to the initial database and before cleaning it we put the right codes (as seen above) and after rerunning the code we see that we have all our 166 countries from the initial dataset.

Code

list_country_GDP <- c(unique(GDPpercapita$code))
setdiff(list_country, list_country_GDP)
#> character(0)

Code

D3_1_GDP_per_capita <- GDPpercapita
D3_2_Military_Expenditure_Percent_GDP <- MilitaryExpenditurePercentGDP
D3_3_Miliraty_Expenditure_Percent_Gov_Exp <- MiliratyExpenditurePercentGovExp

Here are the first few lines of the cleaned dataset of GDP per capita:

<<<<<<< Updated upstream

=======

>>>>>>> Stashed changes

For this dataset, we went from ??? observations for 68 variables to 3818 observations for 4 varibles.

Here are the first few lines of the cleaned dataset of military expenditures in percentage of GDP:

<<<<<<< Updated upstream

=======

>>>>>>> Stashed changes

For this dataset, we went from ??? observations for 68 variables to 3818 observations for 4 varibles.

Here are the first few lines of the cleaned dataset of military expenditures in percentage of government expenditures:

<<<<<<< Updated upstream

=======

>>>>>>> Stashed changes

2.3.4 Dataset on internet usage

To prepare the dataset on internet usage in the world to be merge with the other data, we first, import the data. Then, we keep only the year that we are interested in (2000 to 2022). We also rename the column and keep only the country that match the list of the countries in the main dataset on the SDG.

Code

D4_0_Internet_usage <- read.csv(here("scripts", "data", "InternetUsage.csv")) %>%
  filter(Year >= 2000, Year <= 2022) %>%
  rename(
    code = Code,
    country = Entity,
    year = Year,
    internet_usage = Individuals.using.the.Internet....of.population.
  ) %>%
  mutate(internet_usage = internet_usage / 100) %>%
  filter(code %in% list_country) %>%
  select(code, country, year, internet_usage)

Here are the first few lines of the cleaned dataset of internet usage:

<<<<<<< Updated upstream

=======

>>>>>>> Stashed changes

For this first dataset, we reduced the size from 6,570 observations across 4 variables to 3,433 observations for 4 variables.

2.3.5 Dataset on human freedom index

After importing the data from the CATO Institute website, we noticed that even if the file was called “Human Freedom Index 2022”, the available observations were only going from 2000 up to 2020. We have decided first to modify it in order to match our other datasets, by renaming/encoding/standardizing the columns containing the country names.

Code

data <- read.csv(here("scripts", "data", "human-freedom-index-2022.csv"))

#data in tibble 
datatibble <- tibble(data)

# Rename the column countries into country to match the other datbases
names(datatibble)[names(datatibble) == "countries"] <- "country"

# Make sure the encoding of the country names are UTF-8
datatibble$country <- iconv(datatibble$country, to = "UTF-8", sub = "byte")

# standardize country names
datatibble <- datatibble %>%
  mutate(country = countrycode(country, "country.name", "country.name"))

Once done, we could verify which countries were or were not present between these observations and our main SDG dataset. We have decided to keep the ones that were matching between the two datasets.

Code

# Merge by country name
datatibble <- datatibble %>%
  left_join(D1_0_SDG_country_list, by = "country")

datatibble <- datatibble %>% filter(code %in% list_country)
(length(unique(datatibble$code)))
#> [1] 159

# See which ones are missing
list_country_free <- c(unique(datatibble$code))
setdiff(list_country, list_country_free)
#> [1] "AFG" "CUB" "MDV" "STP" "SSD" "TKM" "UZB"

# Turkey was missing but present in the initial database (it was a problem when stadardizing the country names of D1_0SDG_country_list that we corrected) and the other missing countries are:"AFG" "CUB" "MDV" "STP" "SSD" "TKM" "UZB" 
D5_0_Human_freedom_index <- datatibble

Then, we noticed that there were a lot of columns that were not important for us, as we had 141 variables taken into account. So we have decided to keep the ones that refers to the countries informations (such as code, year, ..) and their human freedom scores per category (pf for personnal freedom, ef for economical freedom).

Code

# erasing useless columns to keep only the general ones. 
D5_0_Human_freedom_index <- select(D5_0_Human_freedom_index, year, country, region, hf_score, pf_rol, pf_ss, pf_movement, pf_religion, pf_assembly, pf_expression, pf_identity, pf_score, ef_government, ef_legal, ef_money, ef_trade, ef_regulation, ef_score, code)

D5_0_Human_freedom_index <- D5_0_Human_freedom_index %>%
  rename(
    pf_law = names(D5_0_Human_freedom_index)[5],      # Renames the 5th column to "pf_law"
    pf_security = names(D5_0_Human_freedom_index)[6]  # Renames the 6th column to "pf_security"
  )

Here are the first few lines of the partialy cleaned dataset on Human Freedom Index scores:

<<<<<<< Updated upstream

=======

>>>>>>> Stashed changes

For this first dataset, we reduced the size from 3’465 observations across 141 variables to 3339 observations for 4 variables.

2.3.6 Dataset on Disasters

For this dataset concerning the Disasters we imported the data from Kaggle as we couldn’t find the original dataset that is private coming from the EOSDIS SYSTEM, an interactive interface for browsing full-resolution, global, daily satellite images from NASA. Once we made sure that our file called “Disasters” was convert into a data frame, we selected some specific columns that we where interested in.

Code

Disasters <- as.data.frame(read.csv(here("scripts", "data", "Disasters.csv"))) %>%
  select(Year, Country, ISO, Location, Continent, Disaster.Subgroup, Disaster.Type, Total.Deaths, No.Injured, No.Affected, No.Homeless, Total.Affected, Total.Damages...000.US..)

Because we knew that our file showed all the disasters in each country over the years (between 1970-2021) and that we wanted to focus on a specific period, we filtered our data to show the years between 2000 and 2022. Then we rearranged our data, changing the data types of all the columns and their names in order to match our other datasets.

Code

# Rearrange the columns, changed the type of data, renamed the columns
Rearanged_Disasters <- Disasters %>%
  filter(Year >= 2000 & Year <= 2022) %>%
  mutate(
    code = as.character(ISO),
    country = as.character(Country),
    year = as.integer(Year),
    continent = as.character(Continent),
    disaster.subgroup = as.character(Disaster.Subgroup),
    disaster.type = as.character(Disaster.Type),
    location = as.character(Location),
    total.deaths = as.numeric(Total.Deaths),
    no.injured = as.numeric(No.Injured),
    no.affected = as.numeric(No.Affected),
    no.homeless = as.numeric(No.Homeless),
    total.affected = as.numeric(Total.Affected),
    total.damages = as.numeric(Total.Damages...000.US..)
  )

We then grouped the data by “year”, “code”, “country” and “continent” and summarize the data. Here you can see that we re-selected specific columns as we saw that our first pre-selection was still too wide and some variables as the disaster.subgroup and disaster.type weren’t pertinent.We arranged the columns based on “code,” “country,” “year,” and “continent” to match the other datasets.

Code

Disasters <- Rearanged_Disasters %>%
  group_by(year,code, country, continent) %>%
  summarize(
    total_deaths = sum(total.deaths, na.rm = TRUE),
    no_injured = sum(no.injured, na.rm = TRUE),
    no_affected = sum(no.affected, na.rm = TRUE),
    no_homeless = sum(no.homeless, na.rm = TRUE),
    total_affected = sum(total.affected, na.rm = TRUE),
    total_damages = sum(total.damages, na.rm = TRUE)
  ) 

D6_0_Disasters <- Disasters %>%
  select(code, country, year, continent, total_deaths, no_injured, no_affected, no_homeless, total_affected, total_damages) %>%
  arrange(code, country, year, continent)

Finally we filtered our disasters data to keep only the countries that are present in our main dataset. We analysed the missing countries and identified three countries (BHR, BRN, MLT) that are unexpectedly missing.

Code

D6_0_Disasters <- D6_0_Disasters %>% filter(code %in% list_country)
length(unique(D6_0_Disasters$code))
#> [1] 163

# Here we see which countries are missing
list_country_disasters <- c(unique(D6_0_Disasters$code))
setdiff(list_country, list_country_disasters)
#> [1] "BHR" "BRN" "MLT"

Here are the first few lines of the cleaned dataset on Disasters:

<<<<<<< Updated upstream

=======

>>>>>>> Stashed changes

2.3.7 Dataset on COVID

This dataset contains information on the COVID19 pandemic between 2020 and 2022. The observation are by year, month, day. After importing the database, we transform the date in format YYYY-MM-DD in order to only keep the year.

Code

COVID <- read.csv(here("scripts", "data", "COVID.csv")) %>%
  select(iso_code, location, date, new_cases_per_million, new_deaths_per_million, stringency_index) %>%
  mutate(date = as.integer(year(date)))

We perform a first round of investigation of the missing values before aggregating the values by year. We begin with the variables “cases per million” and “deaths per million”: seeing that for each country, we have either only missing values, either a very low percentage of missing values (~1%), we can compute the sum over each year and ignore the missing values without altering the data. Indeed, where all the values are missing, the computation will return a NA. We then look at the “stringency” variable and we have 3 scenarios:

1. ~20% of missing values: we ignore missing values when computing the mean to have an idea of stringency each year (because we compute the mean stringency over the year, if some days are missing, it is not a problem, it can not evoluate that fast).

2. all are missing: we can ignore the missing values when computing the mean, because it will still return a missing value

3. almost all are missing: here the mean doesn’t make sense -> we will replace the values by NAs to be coherent. The countries with this issues are: ERI, GUM, PRI and VIR. We verify if they are in our main dataset and since none of these countries are, we can ignore the issue, the lines will be remove later anyway.

We aggregate the observations of all days of a year in one observation per country using the mean.

Code

COVID1 <- COVID %>%
  group_by(iso_code) %>%
  summarize(NaDeaths = round(mean(is.na(new_deaths_per_million)),3),
            NaCases = round(mean(is.na(new_cases_per_million)), 3),
            NaStringency = round(mean(is.na(stringency_index)), 3)) %>%
  pivot_longer(cols = starts_with("Na"), names_to = "Variable", values_to = "NaValue")%>%
  filter(NaValue!=0)

issue_list <- c("ERI", "GUM", "PRI", "VIR")
is.element(issue_list, list_country)
#> [1] FALSE FALSE FALSE FALSE

COVID <- COVID %>%
  group_by(location, date) %>%
  mutate(
    cases_per_million = sum(new_cases_per_million, na.rm = TRUE),
    deaths_per_million = sum(new_deaths_per_million, na.rm = TRUE),
    stringency = mean(stringency_index, na.rm = TRUE)
  )%>%
  ungroup()

Now that all the variables of interest are aggregated by year, we remove all the variables that we don’t need and rename all the remaining variables to match the main dataset.

Code

COVID <- COVID %>%
  group_by(location, date) %>%
  distinct(date, .keep_all = TRUE) %>%
  ungroup()

COVID <- COVID %>% select(-c(new_cases_per_million, new_deaths_per_million, stringency_index))

colnames(COVID) <- c("code", "country", "year", "cases_per_million", "deaths_per_million", "stringency")

We remove the years that exceed 2022, we make sure that the country codes are all iso codes with 3 letters (we observe that sometimes they are preceded by “OWID_”) and we standardize the country codes.

Code

COVID <- COVID[COVID$year <= 2022, ]

COVID$code <- gsub("OWID_", "", COVID$code)

COVID$code <- countrycode(
  sourcevar = COVID$code,
  origin = "iso3c",
  destination = "iso3c"
)

We remove the observations of countries that aren’t in our main dataset on SDGs and find that all the 166 countries that we have in the main SDG dataset are also in this one.

Code

D7_0_COVID <- COVID %>% filter(code %in% list_country)
length(unique(COVID$code))
#> [1] 238

Here are the first few lines of the cleaned dataset on COVID19:

<<<<<<< Updated upstream

=======

>>>>>>> Stashed changes

2.3.8 Dataset on Conflicts

For our conflicts dataset, we imported the data from “The World Banck” data catalog. Once we made sure that our file called “Disasters” was convert into a data frame, we selected some specific columns that we where interested in.

Code

Conflicts <- read.csv(here("scripts", "data", "Conflicts.csv")) %>%
  as.data.frame() %>%
  select(year, country, ongoing, gwsum_bestdeaths, pop_affected, 
         peaceyearshigh, area_affected, maxintensity, maxcumulativeintensity)

Our file showed all the Conflicts and consequences per country over the years (between 2000-2016). We couldn’t find a better and more complete dataset, As we consider conflicts as events, we will only take into account results between 2000 and 2016. Then we rearranged our data, changing the data types of all the columns and their names in order to match our other datasets. We grouped the data by ” year”, “country”, re-selected some variables and summarize the data.

Code

Rearanged_Conflicts <- Conflicts %>%
  filter(year >= 2000 & year <= 2022)%>%
  mutate(
    ongoing = as.integer(ongoing),
    country = as.character(country),
    year = as.integer(year),
    gwsum_bestdeaths = as.numeric(gwsum_bestdeaths),
    pop_affected = as.numeric(pop_affected),
    area_affected = as.numeric(area_affected),
    maxintensity = as.numeric(maxintensity),
    )

# Group the data by "year", "country" and summarize the data
Conflicts <- Rearanged_Conflicts %>%
  group_by(year, country) %>%
  summarize(
    ongoing = sum (ongoing, na.rm = TRUE),
    sum_deaths = sum(gwsum_bestdeaths, na.rm = TRUE),
    pop_affected = sum(pop_affected, na.rm = TRUE),
    area_affected = sum(area_affected, na.rm = TRUE),
    maxintensity = sum(maxintensity, na.rm = TRUE),
  )

After we Selected specific columns from the summarized data and arrange the data by our specified columns. To make our dataset compatible with the main one and let the merging face succeed, we dd some adjustment concerning the country names’ to ensure the compatibility. Then we standardize and merge by country names to finally rearrange the data to retain only the countries present in our main dataset. Note that in the end we can see that only one country is missing that wasn’t in the initial conflicts database: BLR

Code

conflicts <- Conflicts %>%
  select(country, year, ongoing, sum_deaths, pop_affected, area_affected, maxintensity) %>%
  arrange(country, year)

conflicts$country <- iconv(conflicts$country, to = "UTF-8", sub = "byte")

conflicts <- conflicts %>%
  mutate(country = countrycode(country, "country.name", "country.name"))

conflicts <- conflicts %>%
  left_join(D1_0_SDG_country_list, by = "country")

conflicts <- conflicts %>%
  select(code, country, year, ongoing, sum_deaths, pop_affected, area_affected, maxintensity) %>%
  arrange(code, country, year)


D8_0_Conflicts <- conflicts %>% filter(code %in% list_country)
(length(unique(conflicts$code)))
#> [1] 166

# See which countries are missing
list_country_conflicts <- c(unique(conflicts$code))
setdiff(list_country, list_country_conflicts)
#> [1] "BLR"

Here are the first few lines of the cleaned dataset on Conflicts:

<<<<<<< Updated upstream

=======

>>>>>>> Stashed changes

2.3.9 Merging our dataset

By merging our eight pre-cleaned datasets, we create a final database.

Code

D2_1_Unemployment_rate$country <- NULL
merge_1_2 <- D1_0_SDG |> left_join(D2_1_Unemployment_rate, join_by(code, year))

D3_1_GDP_per_capita$country <- NULL
merge_12_3 <- merge_1_2 |> left_join(D3_1_GDP_per_capita, join_by(code, year))

D3_2_Military_Expenditure_Percent_GDP$country <- NULL
merge_12_3 <- merge_12_3 |> left_join(D3_2_Military_Expenditure_Percent_GDP, join_by(code, year)) 

D3_3_Miliraty_Expenditure_Percent_Gov_Exp$country <- NULL
merge_12_3 <- merge_12_3 |> left_join(D3_3_Miliraty_Expenditure_Percent_Gov_Exp, join_by(code, year)) 

D4_0_Internet_usage$country <- NULL
merge_123_4 <- merge_12_3 |> left_join(D4_0_Internet_usage, join_by(code, year)) 

D5_0_Human_freedom_index$country <- NULL
merge_1234_5 <- merge_123_4 |> left_join(D5_0_Human_freedom_index, join_by(code, year)) 

D6_0_Disasters$country <- NULL
merge_12345_6 <- merge_1234_5 |> left_join(D6_0_Disasters, join_by(code, year)) 

D7_0_COVID$country <- NULL
D7_0_COVID <- D7_0_COVID |> distinct(code, year, .keep_all = TRUE)
merge_123456_7 <- merge_12345_6 |> left_join(D7_0_COVID, join_by(code, year)) 

D8_0_Conflicts$country <- NULL
all_Merge <- merge_123456_7 |> left_join(D8_0_Conflicts, join_by(code, year)) 

2.3.10 Cleaning of the final database

2.3.10.1 FILLING COLOMNS

When we merged our dataset, we noticed that some countries were not assigned their corresponding continents and/or region. This issue arose because we sourced the continent and region data from secondary databases, not from our main one. We now add this the corresponding missing continents and regions.

Code

#### Filling missing continents and regions ####

# Update all_Merge with region and continent information
all_Merge <- all_Merge %>%
  group_by(country) %>%
  mutate(
    continent = ifelse(is.na(continent), first(na.omit(continent)), continent),
    region = ifelse(is.na(region), first(na.omit(region)), region)
    ) %>%
  ungroup() %>%
  mutate(continent = case_when(
    code %in% c("BHR") ~ "Asia",
    code %in% c("BRN") ~ "Asia",
    code %in% c("MLT") ~ "Europe",
      TRUE ~ continent
    ))

# Load Disasters dataset to add region information
Disasters <- read.csv(here("scripts", "data", "Disasters.csv")) %>%
  select(ISO, Region) %>%
  distinct(ISO, Region, .keep_all = TRUE) %>%
  rename(code = ISO, region = Region)
# Merge All_Merge with Disasters dataset
all_Merge <- left_join(all_Merge, Disasters, by = "code") %>%
  mutate(region = ifelse(is.na(region.x), region.y, region.x)) %>%
  select(-region.x, -region.y)

We order the database, beginning by the information on the country, the year, the continent and the region.

Code

all_Merge <- as.data.frame(all_Merge) %>%
  select(code, year, country, continent, region, everything())

write.csv(all_Merge, file = here("scripts","data","all_Merge.csv"))

Here are the first few lines of the final dataset:

<<<<<<< Updated upstream

=======

>>>>>>> Stashed changes

Final structure of our merged database: each country of the 166 countries from D1_1_SDG are observed each year from 2000 to 2022, thus each row has a key composed of (code, year) that uniquely identifies an observation. The other columns are the variables listed above. Due to some countries having a lot of missing information we will have to eliminate some of them, but we will still have more than 2000 rows in our database.

2.3.11 Treatment of missing values

We load our final database and we visualize the missing values.

Code

all_Merge <- read.csv(here("scripts","data","all_Merge.csv"))

# Remove unnecessary column
all_Merge <- all_Merge %>% select(-c(X))

# Create a dataframe with the goals without NAs summarize in one column to simplify the visualization
goal_vars <- all_Merge %>%
  select(starts_with("goal")) %>%
  filter_all(all_vars(!is.na(.))) %>%
  colnames()
to_plot_missing <- all_Merge %>%
  mutate(Goals_without_NAs = rowSums(!is.na(select(., all_of(goal_vars))))) %>%
  select(-c(goal2, goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal11, goal12, goal13, goal15, goal16, goal17))

vis_dat(to_plot_missing, warn_large_data = FALSE) + scale_fill_brewer(palette = "Paired") +
  theme(
    axis.text.x = element_text(angle = 90, size = 6),
    legend.text = element_text(size = 8),  # Adjust the size of legend text
    legend.title = element_text(size = 10) 
  )

For each of our research question, we will start with the merged data set and deal with the missing value separately. This will allow us to not delete observations when we do not need to.

For question 1, we only keep the years until 2020, because most of the explanatory variables that we want to use (those coming from the human freedom index) only have values until 2020.

Code

data_question1 <- all_Merge %>%
  filter(year<=2020) %>%
  select(-c(total_deaths, no_injured, no_affected, no_homeless, total_affected, total_damages, cases_per_million, deaths_per_million, stringency, ongoing, sum_deaths, pop_affected, area_affected, maxintensity))

For question 2 and 4, we use the main data from the SDG database.

Code

data_question24 <- all_Merge %>%
  select(c(code, year, country, continent, region, overallscore, goal1, goal2, goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal10, goal11, goal12, goal13, goal15, goal16, goal17))

For question 3, we create 3 distinct databases according to the different type of event that we will analyse: disasters, COVID19 and conflicts. For the disasters, we only keep the years until 2021, because after this date, we don’t have data, moreover we decided to delete the country Bahrain, Brunei and Malta as we do not have any data concerning them. For the conflicts, we only keep the years until 2016, because after this date, we don’t have data. Concerning the conflict dataset, we decided to erase Belarus because once again we do not have any data concerning this country.

Code

# Disasters
data_question3_1 <- all_Merge %>%
  filter(year<=2021 & code!="BHR" & code!="BRN" & code!="MLT") %>%
  select(c(code, year, country, continent, region, overallscore, goal1, goal2, goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal10, goal11, goal12, goal13, goal15, goal16, goal7, total_deaths, no_injured, no_affected, no_homeless, total_affected, total_damages))

# COVID
data_question3_2 <- all_Merge %>%
  select(c(code, year, country, continent, region, overallscore, goal1, goal2, goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal10, goal11, goal12, goal13, goal15, goal16, goal7, cases_per_million, deaths_per_million, stringency))

# Conflicts 
data_question3_3 <- all_Merge %>%
  filter(year<=2016 & code !="BLR") %>%
  select(c(code, year, country, continent, region, overallscore, goal1, goal2, goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal10, goal11, goal12, goal13, goal15, goal16, goal7, ongoing, sum_deaths, pop_affected, area_affected, maxintensity))

2.3.11.1 Data for question 1

2.3.11.1.1 Dealing with missing values in colomns

We begin by visualizing the missing values. To have a less messy graph we group all the goals wihtout NAs into one single variable. We decide to remove MilitaryExpenditurePercentGovExp, because it has too many missing values and it contains similar information to MilitaryExpenditurePercentGDP.We also remove hf_score, pf_score and ef_score, because there are many missing values and since these variables summarize the other ones, deleting them will not make us loose information.

Code

# Create a dataframe with the goals without NAs summarize in one column to simplify the visualization
variable_names <- names(data_question1)
missing_percentages <- sapply(data_question1, function(col) mean(is.na(col)) * 100)

missing_data_summary <- data.frame(
  Variable = variable_names,
  Missing_Percentage = missing_percentages
)

missing_data_summary <- missing_data_summary %>%
  mutate(VariableGroup = ifelse(startsWith(Variable, "goal") & Missing_Percentage == 0, "Goals without NAs", as.character(Variable)))

ggplot(data = missing_data_summary, aes(x = reorder(VariableGroup, Missing_Percentage), y = Missing_Percentage, fill = Missing_Percentage)) +
  geom_bar(stat = "identity") +
  geom_text(aes(label = ifelse(Missing_Percentage > 1, sprintf("%.1f%%", Missing_Percentage), ""),
                y = Missing_Percentage),
            position = position_stack(vjust = 1),  # Adjust vertical position
            color = "white",  # Text color
            size = 2,          # Text size
            hjust = 1.05) +
  labs(title = "Percentage of Missing Values by Variable",
       x = "Variable",
       y = "Missing Percentage") +
  theme_minimal() +
  theme(axis.text.y = element_text(hjust = 1, size=6 ),
        legend.text = element_text(size = 8),
        legend.title = element_text(size = 10)) +
  labs(fill = "% NAs") +
  coord_flip()

data_question1 <- data_question1 %>% select(-c(MiliratyExpenditurePercentGovExp, hf_score, pf_score, ef_score))

2.3.11.1.2 Dealing with missing vlaues in rows

We create a column with the number of missing values by country over all the variables, except goal 1 and goal 10 that we already discussed. We decide to remove the countries that have more than 50 missing values.

Code

see_missing1_1 <- data_question1 %>%
  group_by(code) %>%
  summarise(across(-c(year, country, continent, region, population, overallscore, goal1, goal2, goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal10, goal11, goal12, goal13, goal15, goal16, goal17), 
                   ~ sum(is.na(.))) %>%
              mutate(num_missing = rowSums(across(everything()))) %>%
              filter(num_missing > 50))

data_question1 <- data_question1 %>% filter(!code %in% see_missing1_1$code)

Here is the graph that allows us to visualize the countries that have missing values, how many and for which variables, when there are more than 50 NAs in total.

Code

ggplot(see_missing1_1, aes(x = num_missing , y = reorder(code, num_missing), fill = num_missing)) +
    geom_bar(stat = "identity") + 
    scale_fill_gradient(low = "lightgreen", high = "darkgreen") +
    theme_minimal() +
  theme(axis.text.y = element_text(hjust = 1, size=8 ),
        legend.text = element_text(size = 8),
        legend.title = element_text(size = 10)) +
    labs(title = "Number of missing values per country containing at least 50 NAs", x = "Number of Missing Values", y = "Countries")

Code

see_missing1_2 <- data_question1 %>%
  group_by(code) %>%
  summarise(across(-c(year, country, continent, region, population, overallscore, goal1, goal2, goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal10, goal11, goal12, goal13, goal15, goal16, goal17),
                   ~ sum(is.na(.))) %>%
              mutate(num_missing = rowSums(across(everything()))) %>%
              filter(num_missing > 0))

Here is the ggplot that helps us to visualize the countries that have missing values after removing the countries with more than 50 NAs.

Code

ggplot(see_missing1_2, aes(x = num_missing , y = reorder(code, num_missing), fill = num_missing)) +
    geom_bar(stat = "identity", width = 0.5) + 
    scale_fill_gradient(low = "lightgreen", high = "darkgreen") +
    theme_minimal() +
  theme(axis.text.y = element_text(hjust = 1, size= 6 ),
        legend.text = element_text(size = 8),
        legend.title = element_text(size = 10)) +
        labs(title = "Number of missing values per country", x = "Number of Missing Values", y = "Countries")

We also look at patterns of missing values in the rows and see that except for the two goals with NAs that we discussed earlier and for the triplet “ef_money”, “ef_trade” and “ef_regulation” there are not well defined patterns. We removes the countries that have NAs in the three variables mentioned at the same time.

Code

naniar::gg_miss_upset(data_question1, nsets=10, nintersects=11)

data_question1 <- data_question1[rowSums(is.na(data_question1[, c("ef_money", "ef_trade", "ef_regulation")])) < 3, ]

data_question1 <- data_question1 %>%
  group_by(code) %>%
  filter(all(2000:2020 %in% year)) %>%
  ungroup()

2.3.11.1.3 GDP per capita

Only Venezuela has missing values that we can not fill (because the evolution over time is not linear), so we delete the country.

Code

question1_missing_GDP <- data_question1 %>%
  group_by(code) %>%
  summarize(NaGDPpercapita = mean(is.na(GDPpercapita)))%>%
  filter(NaGDPpercapita != 0)

data_question1 <- data_question1 %>% filter(code!="VEN")

2.3.11.1.4 Military expenditure in % of GDP

For MilitaryExpenditurePercentGDP, We plot the evolution of MilitaryExpenditurePercentGDP along the years for each country containing missing values and distinguish the percentage of missing values with colors.

Code

MilitaryExpenditurePercentGDP1 <- data_question1 %>%
  group_by(code) %>%
  summarize(NaMil1 = round(mean(is.na(MilitaryExpenditurePercentGDP)),3)) %>%
  filter(NaMil1 != 0)

filtered_data_Mil1 <- MilitaryExpenditurePercentGDP %>%
  filter(code %in% MilitaryExpenditurePercentGDP1$code) # countries with NAs

filtered_data_Mil1 <- filtered_data_Mil1 %>%
  group_by(code) %>%
  mutate(PercentageMissing = mean(is.na(MilitaryExpenditurePercentGDP))) %>% # Column % NAs
  ungroup()

Evol_Missing_Mil1 <- ggplot(data = filtered_data_Mil1) +
  geom_line(aes(x = year, y = MilitaryExpenditurePercentGDP, 
                 color = cut(PercentageMissing,
                             breaks = c(0, 0.1, 0.2, 0.3, 1),
                             labels = c("0-10%", "10-20%", "20-30%", "30-100%")))) +
  labs(title = "Military expenditure in % of GDP over time", x = "Years from 2000 to 2022", y = "GDP per capita") +
  scale_color_manual(values = c("0-10%" = "blue", "10-20%" = "green", "20-30%" = "red", "30-100%" = "black"),
                     labels = c("0-10%", "10-20%", "20-30%", "50-100%")) +
  guides(color = guide_legend(title = "% missings")) +
  facet_wrap(~ code, nrow = 5) +
  theme(strip.text = element_text(size = 6)) +
  scale_x_continuous(breaks = NULL) +
  scale_y_continuous(breaks = NULL)

print(Evol_Missing_Mil1)

We delete the countries with more than 30% of missing values and for the countries with less than 30% of missing values and a linear evolution in time, we fill the missing values using linear interpolation.

Code

data_question1 <- data_question1 %>% filter(code!="ARE" & code!="BHS" & code!="BRB" & code!="CRI" & code!="HTI" & code!="ISL" & code!="PAN" & code!="SYR" & code!="VNM") 

list_code <- c("BDI", "BEN", "CAF", "CIV", "COD", "GAB", "NER", "TGO", "TTO", "ZMB")

for (i in list_code) {
  country_data <- data_question1 %>% filter(code == i)
  interpolated_data <- na.interp(country_data$MilitaryExpenditurePercentGDP)
  data_question1[data_question1$code == i, "MilitaryExpenditurePercentGDP"] <- interpolated_data
}

Then, we look at the distribution of the variable per region. Seeing that all are skewed distributions, we decide to replace the remaining missing values, where there are less than 30% missing using the median by region.

Code

question1_missing_Military <- data_question1 %>%
  group_by(code) %>%
  mutate(PercentageMissing = mean(is.na(MilitaryExpenditurePercentGDP))) %>% # Column % NAs
  ungroup() %>%
  group_by(region) %>%
  filter(sum(PercentageMissing, na.rm = TRUE) > 0)

Freq_Missing_Military <- ggplot(data = question1_missing_Military) +
  geom_histogram(aes(x = MilitaryExpenditurePercentGDP, 
                     fill = cut(PercentageMissing,
                                breaks = c(0, 0.1, 0.2, 0.3, 1),
                                labels = c("0-10%", "10-20%", "20-30%", "30-100%"))),
                 bins = 30) +
  labs(title = "Distribution of Military expenditures in % of GDP", x = "Military expenditures in % of GDP", y = "Frequency") +
  scale_fill_manual(values = c("0-10%" = "blue", "10-20%" = "green", "20-30%"="red","30-100%" = "black"), labels = c("0-10%", "10-20%", "20-30%","30-100%")) +
  guides(fill = guide_legend(title = "% missings")) +
  facet_wrap(~ region, nrow = 1)

print(Freq_Missing_Military)

data_question1 <- data_question1 %>%
  group_by(code) %>%
  mutate(
    PercentageMissingByCode = mean(is.na(MilitaryExpenditurePercentGDP))
  ) %>%
  ungroup() %>%  
  group_by(region) %>%
  mutate(
    MedianByRegion = median(MilitaryExpenditurePercentGDP, na.rm = TRUE),
    MilitaryExpenditurePercentGDP = ifelse(
      PercentageMissingByCode < 0.3 & !is.na(MilitaryExpenditurePercentGDP),
      MilitaryExpenditurePercentGDP,
      ifelse(PercentageMissingByCode < 0.3, MedianByRegion, MilitaryExpenditurePercentGDP)
    )
  ) %>%
  select(-PercentageMissingByCode, -MedianByRegion)

2.3.11.1.5 Internet usage

There are only low percentage of missing values.

Code

question1_missing_Internet <- data_question1 %>%
  group_by(code) %>%
  summarize(NaInternet = mean(is.na(internet_usage)))%>%
  filter(NaInternet != 0)

There are never more than 30% of NAs. We look at the evolution of the variable over time. We fill the missing values with linear interpolation, because all are increasing in time and they are almost straight lines, except for CIV that we delete.

Code

question1_missing_Internet <- data_question1 %>%
  group_by(code) %>%
  mutate(PercentageMissing = mean(is.na(internet_usage))) %>% # Column % NAs
  filter(code %in% question1_missing_Internet$code)

Evol_Missing_Internet <- ggplot(data = question1_missing_Internet) +
  geom_line(aes(x = year, y = internet_usage, 
                 color = cut(PercentageMissing,
                             breaks = c(0, 0.1, 0.2, 0.3, 1),
                             labels = c("0-10%", "10-20%", "20-30%", "30-100%")))) +
  labs(title = "Evolution of internet usage over time", x = "Years from 2000 to 2022", y = "Internet usage") +
  scale_color_manual(values = c("0-10%" = "blue", "10-20%" = "green", "20-30%" = "red", "30-100%" = "black"),
                     labels = c("0-10%", "10-20%", "20-30%", "50-100%")) +
  guides(color = guide_legend(title = "% missings")) +
  scale_x_continuous(breaks=NULL)+
  facet_wrap(~ code, nrow = 4)

print(Evol_Missing_Internet)

list_code <- setdiff(unique(question1_missing_Internet$code), "CIV")
for (i in list_code) {
  country_data <- data_question1 %>% filter(code == i)
  interpolated_data <- na.interp(country_data$internet_usage)
  data_question1[data_question1$code == i, "internet_usage"] <- interpolated_data
}

data_question1 <- data_question1 %>% filter(code!="CIV")

2.3.11.1.6 Human freedom index

2.3.11.1.6.1 Personal freedom: law

The variable pf_law has (many) NAs, but only for one country: BLZ, so we decide to remove it.

Code

data_question1 <- data_question1 %>%
  filter(code!="BLZ")

2.3.11.1.6.2 Economic freedom: government

There are no more missing values, thanks to the previous steps.

2.3.11.1.6.3 Economic freedom: money

5 countries have missing values, but the percentage of missing values is always below 25%.

Code

question1_missing_ef_money <- data_question1 %>%
  group_by(code) %>%
  summarize(Na_ef_money = mean(is.na(ef_money)))%>%
  filter(Na_ef_money != 0)

We look at the evolution of the variable over time, and for the countries with a linear evolution in time, we fill the missing values using linear interpolation.

Code

question1_missing_ef_money <- data_question1 %>%
  group_by(code) %>%
  mutate(PercentageMissing = mean(is.na(ef_money))) %>% # Column % NAs
  filter(code %in% question1_missing_ef_money$code)

Evol_Missing_ef_money <- ggplot(data = question1_missing_ef_money) +
  geom_line(aes(x = year, y = ef_money, 
                 color = cut(PercentageMissing,
                             breaks = c(0, 0.1, 0.2, 0.3, 1),
                             labels = c("0-10%", "10-20%", "20-30%", "30-100%")))) +
  labs(title = "Evolution of economic freedom: money over time", x = "Years from 2000 to 2022", y = "ef_money") +
  scale_color_manual(values = c("0-10%" = "blue", "10-20%" = "green", "20-30%" = "red", "30-100%" = "black"),
                     labels = c("0-10%", "10-20%", "20-30%", "50-100%")) +
  guides(color = guide_legend(title = "% missings")) +
  facet_wrap(~ code, nrow = 2) +
  scale_y_continuous(limits = c(0, 10))

print(Evol_Missing_ef_money)

list_code <- c("GEO", "MKD")
for (i in list_code) {
  country_data <- data_question1 %>% filter(code == i)
  interpolated_data <- na.interp(country_data$ef_money)
  data_question1[data_question1$code == i, "ef_money"] <- interpolated_data
}

Then, we look at the distribution of the variable per region. Seeing that all are skewed distributions, we decide to replace the missing values using the median by region.

Code

question1_missing_ef_money <- data_question1 %>%
  group_by(code) %>%
  mutate(PercentageMissing = mean(is.na(ef_money))) %>% # Column % NAs
  ungroup() %>%
  group_by(region) %>%
  filter(sum(PercentageMissing, na.rm = TRUE) > 0)

Freq_Missing_ef_money <- ggplot(data = question1_missing_ef_money) +
  geom_histogram(aes(x = ef_money, 
                     fill = cut(PercentageMissing,
                                breaks = c(0, 0.1, 0.2, 0.3, 1),
                                labels = c("0-10%", "10-20%", "20-30%", "30-100%"))),
                 bins = 30) +
  labs(title = "Distribution of economic freedom: money", x = "ef_money", y = "Frequency") +
  scale_fill_manual(values = c("0-10%" = "blue", "10-20%" = "green", "20-30%"="red","30-100%" = "black"), labels = c("0-10%", "10-20%", "20-30%","30-100%")) +
  guides(fill = guide_legend(title = "% missings")) +
  facet_wrap(~ region, nrow = 1)

print(Freq_Missing_ef_money)

data_question1 <- data_question1 %>%
  group_by(code) %>%
  mutate(
    PercentageMissingByCode = mean(is.na(ef_money))
  ) %>%
  ungroup() %>% 
  group_by(region) %>%
  mutate(
    MedianByRegion = median(ef_money, na.rm = TRUE),
    ef_money = ifelse(
      PercentageMissingByCode < 0.3 & !is.na(ef_money),
      ef_money,
      ifelse(PercentageMissingByCode < 0.3, MedianByRegion, ef_money)
    )
  ) %>%
  select(-PercentageMissingByCode, -MedianByRegion)

2.3.11.1.6.4 Economic freedom: trade

6 countries have missing values, but the percentage of missing values is always below 25%.

Code

question1_missing_ef_trade <- data_question1 %>%
  group_by(code) %>%
  summarize(Na_ef_trade = mean(is.na(ef_trade)))%>% # Column % NAs
  filter(Na_ef_trade != 0)

We look at the evolution of the variable over time. For the countries where this evolution is linear, we fill in the missing values using linear interpolation.

Code

question1_missing_ef_trade <- data_question1 %>%
  group_by(code) %>%
  mutate(PercentageMissing = mean(is.na(ef_trade))) %>% # Column % NAs
  filter(code %in% question1_missing_ef_trade$code)

Evol_Missing_ef_trade <- ggplot(data = question1_missing_ef_trade) +
  geom_line(aes(x = year, y = ef_trade, 
                 color = cut(PercentageMissing,
                             breaks = c(0, 0.1, 0.2, 0.3, 1),
                             labels = c("0-10%", "10-20%", "20-30%", "30-100%")))) +
  labs(title = "Evolution of economic freedom: trade over time", x = "Years from 2000 to 2022", y = "ef_trade") +
  scale_color_manual(values = c("0-10%" = "blue", "10-20%" = "green", "20-30%" = "red", "30-100%" = "black"),
                     labels = c("0-10%", "10-20%", "20-30%", "50-100%")) +
  guides(color = guide_legend(title = "% missings")) +
  facet_wrap(~ code, nrow = ) +
  scale_y_continuous(limits = c(0, 10))

print(Evol_Missing_ef_trade)

# Linear interpolation for "AZE", "BFA", "ETH", "GEO", "VNH"
list_code <- c("AZE", "GEO", "MKD", "MNG")
for (i in list_code) {
  country_data <- data_question1 %>% filter(code == i)
  interpolated_data <- na.interp(country_data$ef_trade)
  data_question1[data_question1$code == i, "ef_trade"] <- interpolated_data
}

Then, we look at the distribution of the variable per region. Seeing that the only region that still has missing values is a centered distribution, we decide to replace the missing values using the mean of the region.

Code

question1_missing_ef_trade <- data_question1 %>%
  group_by(code) %>%
  mutate(PercentageMissing = mean(is.na(ef_trade))) %>% # Column % NAs
  ungroup() %>%
  group_by(region) %>%
  filter(sum(PercentageMissing, na.rm = TRUE) > 0)

Freq_Missing_ef_trade <- ggplot(data = question1_missing_ef_trade) +
  geom_histogram(aes(x = ef_trade, 
                     fill = cut(PercentageMissing,
                                breaks = c(0, 0.1, 0.2, 0.3, 1),
                                labels = c("0-10%", "10-20%", "20-30%", "30-100%"))),
                 bins = 30) +
  labs(title = "Distribution of economic freedom: trade", x = "ef_trade", y = "Frequency") +
  scale_fill_manual(values = c("0-10%" = "blue", "10-20%" = "green", "20-30%"="red","30-100%" = "black"), labels = c("0-10%", "10-20%", "20-30%","30-100%")) +
  guides(fill = guide_legend(title = "% missings")) +
  facet_wrap(~ region, nrow = 2)

print(Freq_Missing_ef_trade)

data_question1 <- data_question1 %>%
  group_by(code) %>%
  mutate(
    PercentageMissingByCode = mean(is.na(ef_trade))
  ) %>%
  ungroup() %>% 
  group_by(region) %>%
  mutate(
    MeanByRegion = mean(ef_trade, na.rm = TRUE),
    ef_trade = ifelse(
      PercentageMissingByCode < 0.3 & !is.na(ef_trade),
      ef_trade,
      ifelse(PercentageMissingByCode < 0.3, MeanByRegion, ef_trade)
    )
  ) %>%
  select(-PercentageMissingByCode, -MeanByRegion)

2.3.11.1.6.5 Economic freedom: regulation

There are no more missing values, thanks to the previous steps.

2.3.11.1.7 SDGs 1 and 10

We noticed earlier that there were only missing values for goals 1 and 10. As we did before, we have started to investigate where are located the NAs in our dataset for first goal1, then goal 10.

Code

na_count <- sapply(data_question1, function(x) sum(is.na(x)))
na_count_df <- data.frame(variable = names(na_count), num_missing = na_count)
na_count_df_filtered <- subset(na_count_df, num_missing > 0)
ggplot(na_count_df_filtered, aes(x= num_missing,y=variable, fill = num_missing)) +
    geom_bar(aes(fill = num_missing), stat = "identity", width = 0.8, fill = 'lightblue') +
    geom_text(aes(label = num_missing), vjust = 0.5,hjust = 1.1, position = position_dodge(width = 0.9)) +
    theme_minimal() +
    theme(axis.text.y = element_text(hjust = 1, size=10 ), 
          legend.text = element_text(size = 8),
          legend.title = element_text(size = 10)) +
    labs(title = "Number of remaining missing values per variable ",
         x = "Number of NAs",
         y = "Variables")

# goal1
question1_missing_goal1 <- data_question1 %>%
  group_by(code) %>%
  summarize(Na_goal1 = mean(is.na(goal1)))%>%
  filter(Na_goal1 != 0)

data_question1 <- data_question1 %>% filter(!code %in% question1_missing_goal1$code)
# still 42 NA values goal10

We had found that the missing values were located in only 5 countries. So we have decided to get rid of them. At this stage, there were only 42 remaining missing values. Then, same step for goal 10.

Code

#goal10
question1_missing_goal10 <- data_question1 %>%
  group_by(code) %>%
  summarize(Na_goal10 = mean(is.na(goal10)))%>%
  filter(Na_goal10 != 0)

data_question1 <- data_question1 %>% filter(!code %in% question1_missing_goal10$code)

We have found the 2 lasts countries containing missing values. Now, our dataset is completely clean and ready to be used for our question 1.

2.3.11.2 Data for question 2 and 4

We create a column with the number of missing values by country over all the variables, except goal 1 and goal 10 that we already discussed. Since there are no other missing values, we stop here.

Code

see_missing24 <- data_question24 %>%
  group_by(code) %>%
  summarise(across(everything(), ~ sum(is.na(.))) %>%
              mutate(num_missing = rowSums(across(everything()))) %>%
              filter(num_missing > 0))
#> `summarise()` has grouped output by 'code'. You can override using
#> the `.groups` argument.

data_question24 <- data_question24 %>%
  group_by(country) %>%
  filter(!all(is.na(goal1)) & !all(is.na(goal10)))

2.3.11.3 Data for question 3

We create a column with the number of missing values by country over all the variables, except goal 1 and goal 10 that we already discussed. Since there are no other missing values, we stop here.

Disasters

We begin by visualizing the missing values.

Code

variable_names <- names(data_question3_1)
missing_percentages <- sapply(data_question3_1, function(col) mean(is.na(col)) * 100)

missing_data_summary <- data.frame(
  Variable = variable_names,
  Missing_Percentage = missing_percentages
)

missing_data_summary <- missing_data_summary %>%
  mutate(VariableGroup = ifelse(startsWith(Variable, "goal") & Missing_Percentage == 0, "Goals without NAs", as.character(Variable)))

ggplot(data = missing_data_summary, aes(x = reorder(VariableGroup, Missing_Percentage), y = Missing_Percentage, fill = Missing_Percentage)) +
  geom_bar(stat = "identity") +
  geom_text(aes(label = ifelse(Missing_Percentage > 1, sprintf("%.1f%%", Missing_Percentage), ""),
                y = Missing_Percentage),
            position = position_stack(vjust = 1),  # Adjust vertical position
            color = "white",  # Text color
            size = 3,          # Text size
            hjust = 1.05) +
  labs(title = "Percentage of Missing Values by Variable",
       x = "Variable",
       y = "Missing Percentage") +
  theme_minimal() +
  theme(axis.text.y = element_text(hjust = 1)) +
  coord_flip()

In this particular case, even if there are many missing values in our disaster dataset, we made the hypothesis that disaster events can not happen every year for every country given that these are uncontrollable and non-recurring events. Therefore the NAs that we encounter will become zeroes, implying that there have been no climatic disasters.

Code

data_question3_1[is.na(data_question3_1)] <- 0

COVID19

We look at the missing values for the three variables that are specific to COVID during the years of COVID: 2020 to 2022. We delete the countries that have NAs (only stringency has 6 countries with 100% NAs).

Code

COVID4 <- data_question3_2 %>%
  filter(year >= 2020 & year <= 2022) %>%
  group_by(code) %>%
  summarize(Na_deaths = mean(is.na(deaths_per_million)),
            Na_cases = mean(is.na(cases_per_million)),
            Na_stringency = mean(is.na(stringency))) %>%
  filter(Na_deaths != 0 | Na_cases!=0 |  Na_stringency !=0)

ggplot(COVID4, aes(x = reorder(code, Na_deaths), y = Na_deaths)) +
  geom_bar(stat = "identity", fill = "lightgreen", color = "black") +
  labs(title = "NAs by rows: deaths per million",
       x = "Code",
       y = "Proportion of Missing Values") +
  theme_minimal() +
  theme(axis.text.x = element_text(angle = 45, hjust = 1))

ggplot(COVID4, aes(x = reorder(code, Na_cases), y = Na_cases)) +
  geom_bar(stat = "identity", fill = "lightgreen", color = "black") +
  labs(title = "NAs by rows: cases per million",
       x = "Code",
       y = "Proportion of Missing Values") +
  theme_minimal() +
  theme(axis.text.x = element_text(angle = 45, hjust = 1))

ggplot(COVID4, aes(x = reorder(code, Na_stringency), y = Na_stringency)) +
  geom_bar(stat = "identity", fill = "lightgreen", color = "black") +
  labs(title = "NAs by rows: stringency",
       x = "Code",
       y = "Proportion of Missing Values") +
  theme_minimal() +
  theme(axis.text.x = element_text(angle = 45, hjust = 1))

data_question3_2 <- data_question3_2 %>% filter(!code %in% COVID4$code)

We replace the NAs of the other COVID columns (years 2000 t0 2019) by 0 (because we don’t have real missing, only introduced by merging with the other databases).

Code

data_question3_2 <- data_question3_2 %>%
  mutate(
    cases_per_million = ifelse(is.na(cases_per_million), 0, cases_per_million),
    deaths_per_million = ifelse(is.na(deaths_per_million), 0, deaths_per_million),
    stringency = ifelse(is.na(stringency), 0, stringency)
  )

Conflicts

We create a column with the number of missing values by country over all the variables, except goal 1 and goal 10 that we already discussed.Two countries have missing values, we remove them (MNE and SRB).

Code

see_missing3_3 <- data_question3_3 %>%
  group_by(code) %>%
  summarise(across(-c(goal1, goal10),  # Exclude columns "goal1" and "goal10"
                   ~ sum(is.na(.))) %>%
              mutate(num_missing = rowSums(across(everything()))) %>%
              filter(num_missing > 0))

data_question3_3 <- data_question3_3 %>% filter(!code %in% c("MNE","SRB","SSD"))

##### EXPORT as CSV #####
write.csv(data_question1, file = here("scripts","data","data_question1.csv"))
write.csv(data_question24, file = here("scripts","data","data_question24.csv"))
write.csv(data_question3_1, file = here("scripts","data","data_question3_1.csv"))
write.csv(data_question3_2, file = here("scripts","data","data_question3_2.csv"))
write.csv(data_question3_3, file = here("scripts","data","data_question3_3.csv"))

3 Exploratory data analysis

3.1 Focus on the influence of the factors over the SDG scores

For this first part of our EDA, let’s try to check first the distribution of the variables selected for answering our 1st question.

Code

# Reshape the data from wide to long format for our sdg goals and our human freedom index scores
long_df_goal_distribution <- pivot_longer(Correlation_overall, cols = starts_with("goal"), names_to = "Goal", values_to = "Value")

long_df_hfi_distribution <- pivot_longer(Correlation_overall, cols = pf_law:ef_regulation, names_to = "Category", values_to = "Value")

ggplot(long_df_goal_distribution, aes(x = Value, y = Goal, fill = stat(x))) +
  geom_density_ridges_gradient(scale = 3, size = 0.3, rel_min_height = 0.01) +
  scale_fill_viridis_c(name = "Value", option = "C") +
  labs(title = 'Distribution of Goals')

As we can see, most of our goals have a left-skewed distribution, which shows that for most of the country concerned implemented good strategies for targeting the goals objectives. Some distribution have a longer distribution than other, which could be a proof of inequality in the investments made for implementing solutions. In another hand, we notice that the only right-skewed distribution is concerning the observations of goal 9, which is promoting infrastructures, innovation and inclusive and sustainable industrialization. Again, that could show means inequalities. Wealthier countries are able to invest more on these sustainable development goals. ::: {.cell layout-align=“center” hash=‘report_cache/html/unnamed-chunk-179_d72f8281341cf93c0007f6c43d5b63d8’}

Code

ggplot(long_df_hfi_distribution, aes(x = Value, y = Category, fill = stat(x))) +
  geom_density_ridges_gradient(scale = 3, size = 0.3, rel_min_height = 0.01) +
  scale_fill_viridis_c(name = "Value", option = "C") +
  labs(title = 'Distribution of Human Freedom Index Scores')

::: The distribution of the Human Freedom Index Score follows the same trend. Most of the scores are left-skewed, which means that countries tend to have in general good scores. The only scores not folowing are pf_law and ef_legal, which tend to have lower scores in general. Legal system, for civilians and countries, is changing slowly because it has a lot of implications over the situation within a country/between countries and because of the divergence of opinions. Therefore, investing more money for raising these scores will take more time than raising the scores of other goals.

Now let’s consider the remaining variables of the dataset dedicated to answering the influence of factors over our SDG goals scores. All these variables have right-skewed distribution. Taking the mode into account, most of the concerned countries in our data have an unemployment rate between 2 to 7%, a distribution of GDP per capita between $3’000-$10’000, a distribution of military expenditure in percentage of the GDP 10% to 20% and finally a internet usage between 0 and 10%.

These variables shows us even more the inequalities between the countries in our dataset. While most of our contries have low internet usage or/and a low GDP per capita, few countries are more developed, then mostly wealthier, and thus having better chances to get higher scores. ::: {.cell layout-align=“center” hash=‘report_cache/html/unnamed-chunk-180_e71976e50851b11b573758a2e8488a68’}

Code

#now, same for the remaining variables. No need to reshape our data as only one variable.
unempl_d <- ggplot(Correlation_overall, aes(x = unemployment.rate, y = 1, fill = stat(x))) +
  geom_density_ridges_gradient(scale = 3, size = 0.3, rel_min_height = 0.01) +
  scale_fill_viridis_c(name = "Unemployment Rate", option = "C") +
  labs(title = 'Distribution of Unemployment Rate')

gdp_d <- ggplot(Correlation_overall, aes(x = GDPpercapita, y = 1, fill = stat(x))) +
  geom_density_ridges_gradient(scale = 3, size = 0.3, rel_min_height = 0.01) +
  scale_fill_viridis_c(name = "GDP per Capita", option = "C") +
  labs(title = 'Distribution of GDP per Capita')

milit_d <- ggplot(Correlation_overall, aes(x = MilitaryExpenditurePercentGDP, y = 1, fill = stat(x))) +
  geom_density_ridges_gradient(scale = 3, size = 0.3, rel_min_height = 0.01) +
  scale_fill_viridis_c(name = "Military Expenditure (% of GDP)", option = "C") +
  labs(title = 'Distribution of Military Expenditure (% of GDP)')

internet_d <- ggplot(Correlation_overall, aes(x = internet_usage, y = 1, fill = stat(x))) +
  geom_density_ridges_gradient(scale = 3, size = 0.3, rel_min_height = 0.01) +
  scale_fill_viridis_c(name = "Internet Usage", option = "C") +
  labs(title = 'Distribution of Internet Usage')

grid.arrange(unempl_d,gdp_d,milit_d,internet_d, ncol = 2, nrow = 2)

::: Now, let’s display the distribution of the different SDG achievement scores per continent, using violin boxplots to have an overview of the mods, the range with most of the observations and the outliers. ::: {.cell layout-align=“center” hash=‘report_cache/html/unnamed-chunk-181_018d9323843b464b11703ce77759cd1c’}

Code

#### boxplots ####

#For sdg goals per continent 

#Africa
data_Q1_Africa <- data_question1 %>% #filtering Africa as continent
  filter(data_question1$continent == 'Africa') %>%
  dplyr::select(continent, overallscore, goal1, goal2, goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal10, goal11, goal12, goal13, goal15, goal16, goal17)

data_Q1_Africa_long <- melt(data_Q1_Africa)

medians_AF <- data_Q1_Africa_long %>% #median per variable
  group_by(variable) %>%
  summarize(median_value = median(value))
data_Q1_Africa_long <- data_Q1_Africa_long %>%
                  left_join(medians_AF, by = "variable")
#America
data_Q1_Americas <- data_question1 %>%#filtering Americas as continent
  filter(data_question1$continent == 'Americas') %>%
  dplyr::select(continent, overallscore, goal1, goal2, goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal10, goal11, goal12, goal13, goal15, goal16, goal17)

data_Q1_Americas_long <- melt(data_Q1_Americas)

medians_AM <- data_Q1_Americas_long %>% #median per variable
  group_by(variable) %>%
  summarize(median_value = median(value))
data_Q1_Americas_long <- data_Q1_Americas_long %>%
                  left_join(medians_AM, by = "variable")
#Asia
data_Q1_Asia <- data_question1 %>%
  filter(data_question1$continent == 'Asia') %>%#filtering Asia as continent
  dplyr::select(continent, overallscore, goal1, goal2, goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal10, goal11, goal12, goal13, goal15, goal16, goal17)

data_Q1_Asia_long <- melt(data_Q1_Asia)

medians_AS <- data_Q1_Asia_long %>% #median per variable
  group_by(variable) %>%
  summarize(median_value = median(value))
data_Q1_Asia_long <- data_Q1_Asia_long %>%
                  left_join(medians_AS, by = "variable")
#Europe
data_Q1_Europe <- data_question1 %>%
  filter(data_question1$continent == 'Europe') %>% #filtering Europe as continent
  dplyr::select(continent, overallscore, goal1, goal2, goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal10, goal11, goal12, goal13, goal15, goal16, goal17)

data_Q1_Europe_long <- melt(data_Q1_Europe)

medians_EU <- data_Q1_Europe_long %>% #median per variable
  group_by(variable) %>%
  summarize(median_value = median(value))
data_Q1_Europe_long <- data_Q1_Europe_long %>%
                  left_join(medians_EU, by = "variable")
#Oceania
data_Q1_Oceania <- data_question1 %>%
  filter(data_question1$continent == 'Oceania') %>% #filtering Oceania as continent
  dplyr::select(continent, overallscore, goal1, goal2, goal3, goal4, goal5, goal6, goal7, goal8, goal9, goal10, goal11, goal12, goal13, goal15, goal16, goal17)

data_Q1_Oceania_long <- melt(data_Q1_Oceania)

medians_OC <- data_Q1_Oceania_long %>% #median per variable
  group_by(variable) %>%
  summarize(median_value = median(value))
data_Q1_Oceania_long <- data_Q1_Oceania_long %>%
                  left_join(medians_OC, by = "variable")
# merge all medians
medians_all <- rbind(data_Q1_Oceania_long, data_Q1_Americas_long,data_Q1_Africa_long,data_Q1_Asia_long,data_Q1_Europe_long)

medians_all$color <- ifelse(medians_all$median_value > 75, "lightblue",
                        ifelse(medians_all$median_value < 25, "red", 'orange')) #assigning colors. If median for a goal is > 75 -> lightblue, if < 25 -> red, orange otherwise.

bandwidth_nrd <- bw.nrd(medians_all$value) #adapting the bandwidth

ggplot(medians_all, aes(x = variable, y = value, fill = color)) +
  geom_violin(trim = FALSE, bw = bandwidth_nrd) +
  #geom_boxplot(width = 0.1, outlier.size = 1, fill = 'white') +
  scale_fill_identity() +
  labs(title = "SDG Goals by Region", x = "Goals", y = "Score") +
  facet_grid(continent ~ ., scales = "free_y") +
  scale_y_continuous(labels = scales::label_number()) +
  theme_classic() +
  theme(axis.text.x = element_text(angle = 90, vjust = 0.5, hjust=1))

::: Here is the distribution of the goals per continent. We notice that Europe the continent with most of its goals having a median superior to 75 (represented by the lightblue color. We notice that only two goals have a median score lower than 25, which is for goal 9 for Africa and goal 10 for America. As seen before, goal 9 is generally having lower scores than the other goals. That could mean that the access to technology and sustainable/resilient infrastructures/industrialization is harder in Africa, because of various reasons such as less wealthy countries, corruption,…

The goal 10 concerns the reduction of inequalities within/amongst countries. Therefore, we presume that less effort and investissement has been made on this goal in America.

In addition, some distributions are quite disparsed, such as goal 13 in Oceania and goal 10 in Africa. That could again show inequalities within countries or less investment made to raise the scores by different countriews of the same continent.

Now let’s display boxplots for the different variables of the human freedom index. ::: {.cell layout-align=“center” hash=‘report_cache/html/unnamed-chunk-182_45ef62a2fa67d6acc84dbce0cefc4fc5’}

Code

#for Human Freedom Index scores 

#Africa
data_Q1_Africa_HFI <- data_question1 %>%
  filter(data_question1$continent == 'Africa') %>%
  dplyr::select(continent, pf_law, pf_security, pf_movement, pf_religion, pf_assembly, pf_expression, pf_identity, ef_government, ef_legal, ef_money, ef_trade, ef_regulation)

data_Q1_Africa_HFI_long <- melt(data_Q1_Africa_HFI)

medians_AF_HFI <- data_Q1_Africa_HFI_long %>%
  group_by(variable) %>%
  summarize(median_value = median(value))

data_Q1_Africa_HFI_long <- data_Q1_Africa_HFI_long %>%
                  left_join(medians_AF_HFI, by = "variable")

#America
data_Q1_Americas_HFI <- data_question1 %>%
  filter(data_question1$continent == 'Americas') %>%
  dplyr::select(continent, pf_law, pf_security, pf_movement, pf_religion, pf_assembly, pf_expression, pf_identity, ef_government, ef_legal, ef_money, ef_trade, ef_regulation)

data_Q1_Americas_HFI_long <- melt(data_Q1_Americas_HFI)

medians_AM_HFI <- data_Q1_Americas_HFI_long %>%
  group_by(variable) %>%
  summarize(median_value = median(value))

data_Q1_Americas_HFI_long <- data_Q1_Americas_HFI_long %>%
                  left_join(medians_AM_HFI, by = "variable")

#Asia
data_Q1_Asia_HFI <- data_question1 %>%
  filter(data_question1$continent == 'Asia') %>%
  dplyr::select(continent, pf_law, pf_security, pf_movement, pf_religion, pf_assembly, pf_expression, pf_identity, ef_government, ef_legal, ef_money, ef_trade, ef_regulation)

data_Q1_Asia_HFI_long <- melt(data_Q1_Asia_HFI)

medians_AS_HFI <- data_Q1_Asia_HFI_long %>%
  group_by(variable) %>%
  summarize(median_value = median(value))

data_Q1_Asia_HFI_long <- data_Q1_Asia_HFI_long %>%
                  left_join(medians_AS_HFI, by = "variable")

#Europe
data_Q1_Europe_HFI <- data_question1 %>%
  filter(data_question1$continent == 'Europe') %>%
  dplyr::select(continent, pf_law, pf_security, pf_movement, pf_religion, pf_assembly, pf_expression, pf_identity, ef_government, ef_legal, ef_money, ef_trade, ef_regulation)

data_Q1_Europe_HFI_long <- melt(data_Q1_Europe_HFI)

medians_EU_HFI <- data_Q1_Europe_HFI_long %>%
  group_by(variable) %>%
  summarize(median_value = median(value))

data_Q1_Europe_HFI_long <- data_Q1_Europe_HFI_long %>%
                  left_join(medians_EU_HFI, by = "variable")

#Oceania 
data_Q1_Oceania_HFI <- data_question1 %>%
  filter(data_question1$continent == 'Oceania') %>%
  dplyr::select(continent, pf_law, pf_security, pf_movement, pf_religion, pf_assembly, pf_expression, pf_identity, ef_government, ef_legal, ef_money, ef_trade, ef_regulation)

data_Q1_Oceania_HFI_long <- melt(data_Q1_Oceania_HFI)

medians_OC_HFI <- data_Q1_Oceania_HFI_long %>%
  group_by(variable) %>%
  summarize(median_value = median(value))

data_Q1_Oceania_HFI_long <- data_Q1_Oceania_HFI_long %>%
                  left_join(medians_OC_HFI, by = "variable")

# merge all medians 
medians_all_HFI <- rbind(data_Q1_Oceania_HFI_long, data_Q1_Americas_HFI_long,data_Q1_Africa_HFI_long,data_Q1_Asia_HFI_long,data_Q1_Europe_HFI_long)

medians_all_HFI$color <- ifelse(medians_all_HFI$median_value > 7.5, "lightblue", 
                        ifelse(medians_all_HFI$median_value < 2.5, "red", 'orange'))

bandwidth_nrd_HFI <- bw.nrd(medians_all_HFI$value)

# Create the plot
ggplot(medians_all_HFI, aes(x = variable, y = value, fill = color)) +
  geom_violin(trim = FALSE, bw = bandwidth_nrd_HFI) +
  #geom_boxplot(width = 0.1, outlier.size = 1, fill = 'white') +
  scale_fill_identity() +
  labs(title = "Human Freedom Index Scores by Region", x = "Variables", y = "Score") +
  facet_grid(continent ~ ., scales = "free_y") +
  scale_y_continuous(labels = scales::label_number()) +
  theme_classic() +
  theme(axis.text.x = element_text(angle = 90, vjust = 0.5, hjust=1))

::: Here we can notice the same results as before concerning the SDG goals, except that no score has a median below 25%. Again, Europe is the continent with most of its median scores superior to 75 (lightblue color)

For space reason because of the different scales, we have decided not to make violin boxplot per continent for the remaining variables. The distribution can be seen in the general distribution seen prior to that.

Now, let’s have a closer look to the general correlation between our variables. Using our cleaned dataset, we will use a correlation heatmap to help us vizualising the informations. Given that most of our variables are not normally distributed, we will use the Spearman method to calculate the correlation. ::: {.cell layout-align=“center” hash=‘report_cache/html/unnamed-chunk-183_77635a4458a513ebfb16d76e15cb4451’}

Code

#### Correlations between variables Heatmap ####

Correlation_overall <-data_question1 %>% # selection of the numerical data
      dplyr::select(population:ef_regulation)

cor_matrix_sper <- # calculation of the correlation matrix
  cor(Correlation_overall, method = "spearman", use = "everything")

cor_melted <- # wide to long transformation
  melt(cor_matrix_sper)

ggplot(data = cor_melted, aes(Var1, Var2, fill = value)) +
  geom_tile() +
  scale_fill_gradient2(low = "blue", high = "red", mid = "white", 
                       midpoint = 0, limit = c(-1, 1), space = "Lab", 
                       name="Spearman\nCorrelation") +
  theme_minimal() +
  theme(axis.text.x = element_text(angle = 45, vjust = 1, size = 8, hjust = 1),
        axis.text.y = element_text(size = 8)) +
  coord_fixed() +
  labs(x = '', y = '', title = 'Correlation Matrix Heatmap')

::: By looking at our heatmap, we notice that most of our goals are strongely correlated together and that some variables amongst the Human Freedom Index scores too (strong correlation among personal freedom variables (pf), reflecting scores from the Human Freedom Index on movement, religion, assembly, and expression). This could be explained by the fact that some of these goals/scores share partially similar objectifs, which could mean that a raise in the score of one of these goals will raise positively the score of another/some other goals. In addition, we notice that goals 12 and 13 (respectively “responsible consumption & production” and “climate action”) are strongely negatively correlated with most of our variables, except between themself.

We will see more in detail the correlations between our goals and variables in the analysis part of the influence of the factors over the Sustainable Development Goals.

Code

#### Spearman's correlation coeff ####

# lower.panel <- function(x, y, ...){
#   points(x, y, pch = 20, col = "darkgreen", cex = 0.2)
# }
# 
# panel.hist <- function(x, ...){
#   usr <- par("usr"); on.exit(par(usr))
#   par(usr = c(usr[1:2], 0, 1.5) )
#   h <- hist(x, plot = FALSE)
#   breaks <- h$breaks; nB <- length(breaks)
#   y <- h$counts; y <- y/max(y)
#   rect(breaks[-nB], 0, breaks[-1], y, col = "lightgreen", ...)
# }
# 
# # panel.cor_stars function with stars alongside correlation coefficients
# panel.cor_stars <- function(x, y, digits = 2, prefix = "", cex.cor, ...) {
#   usr <- par("usr"); on.exit(par(usr))
#   par(usr = c(0, 1, 0, 1))
#   r <- cor(x, y)
#   p_value <- cor.test (x,y)$p.value
# 
#   if (p_value < 0.001){
#     stars <- "***"
#   } else if (p_value < 0.01) {
#     stars <- "**"
#   } else if (p_value < 0.05) {
#     stars <- "*"
#   } else {
#     stars <- ""
#   }
#   txt <- paste0(format(c(r, 0.123456789), digits = digits)[1], " ", stars)
#   if(missing(cex.cor)) cex.cor <- 0.5/strwidth(txt)
#   text(0.5, 0.5, txt, cex = cex.cor)
# }
# 
# # # Independent variables
# pairs(Correlation_overall[,c("overallscore", "unemployment.rate", "GDPpercapita", "MilitaryExpenditurePercentGDP", "internet_usage")], upper.panel=panel.cor_stars, diag.panel=panel.hist, lower.panel = lower.panel, main="Correlation table and distribution of various variables")
# 
# pairs(Correlation_overall[,c("overallscore", "pf_law", "pf_security", "pf_movement", "pf_religion", "pf_assembly" ,"pf_expression" ,"pf_identity", "ef_government", "ef_legal", "ef_money", "ef_trade", "ef_regulation")], upper.panel=panel.cor_stars, diag.panel=panel.hist, lower.panel = lower.panel, main="Correlation table and distribution of HFI variables")

<<<<<<< Updated upstream